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ABSTRACT 


This  thesis  contains  an  analysis  of  the  last  five  years  of 
Antisubmarine  Warfare  (ASW)  Weapon  System  Accuracy  Trial  (WSAT) 
data  from  both  the  Atlantic  and  Pacific  Fleet.  The  analysis  is 
conducted  in  an  effort  to  provide  recommendations  to  be  applied 
toward  future  evolution  of  the  ASW  Test  Program  for  surface  ships. 
A  statistical  chi-square  test  is  conducted  on  Fleet  and  Navy  wide 
data  to  determine  which  ASW  combat  system  material  categories  are 
most  prone  to  degradation.  Additionally,  a  critical  examination  of 
the  existing  WSAT  data  base  is  provided  with  an  aim  toward 
promoting  future  statistical  analysis.  Results  of  this  thesis 
indicate  that  degradation  to  weapons  delivery  systems  like  torpedo 
tubes  and  ASROC  launchers  is  statistically  more  significant  than 
the  other  WSAT  test  categories.  The  thesis  also  recommends  new  ways 
to  adapt  the  existing  WSAT  data  base  to  conduct  more  informative 
inspections  of  existing  and  new  construction  ships. 
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I.  INTRODUCTION 


A.  WEAPONS  SYSTEM  ACCURACY  TRIAL  (WSAT)  BACKGROUND 
1.  History 

The  history  of  the  Anti-Submarine  Warfare  (ASW)  Weapon 
System  Test  program  for  surface  ships  dates  back  to  post  World 
War  II  studies  of  ineffective  torpedoes,  sensor  alignment 
errors,  and  fire  control  accuracy  problems.  Presentation 
materials  obtained  from  the  Undersea  Warfare  Museum  in 
Keyport,  Wa.  indicate  initial  systems  level  tests  conducted  on 
Dabob  Bay,  Washington  as  early  as  1959  led  to  the 
establishment  of  WSAT  tests  and  began  the  genesis  of  the 
surface  ship  ASW  test  program.  Presently,  WSAT  is  a  total 
system  test  which  demonstrates  the  ability  of  a  ship's  ASW 
combat  system  to  pass  prescribed  equipment  performance 
standards  tests.  The  total  test  concept  is  shown  in  Figure  1. 

WSAT  -  A  TOTAL  SYSTEM  TEST 


Figure  1.  WSAT  Overview 
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2.  Responsibilities 

Commander  Naval  Seas  Systems  Command  (NAVSEA  06UT)  has 
overall  responsibility  for  WSAT  and  is  the  activity 
responsible  for  surface  ship  ASW  system  certification  and 
recertification.  The  WSAT  results  are  used  by  NAVSEA  as  a 
basis  for  granting  this  certification.  ASW  certification  is 
required  on  every  surface  ship  ASW  combat  system  subsequent  to 
new  construction  and  commissioning.  Similarly,  ASW  system 
recertification  is  required  on  existing  surface  ships  which 
hold  an  ASW  required  operational  capability  upon  exiting 
regular  overhaul  (ROH)  or  upon  receiving  a  major  ASW  system 
suite  upgrade  in  some  other  industrial  period.  Commander 
Naval  Underwater  Systems  Center  (NUSC) ,  Newport,  Rhode  Island 
is  designated  as  the  WSAT  program  technical  direction  agent 
(TDA)  responsible  for  all  WSAT  test  and  analysis  procedures 
and  documentation. 

The  WSAT  organization  supports  both  Atlantic  and 
Pacific  Fleet  units.  Surface  ship  WSAT  inspections  on  the 
East  coast  are  conducted  by  NUSC  technicians  at  either  the 
Atlantic  Fleet  Weapons  Training  Facility  (AFWTF)  3-D 
underwater  range  near  Ft.  Croix  with  dockside  tests  at  Naval 
Station  Roosevelt  Roads,  Puerto  Rico  or  at  the  Atlantic 
Undersea  Test  and  Evaluation  Center  (AUTEC)  3-D  range  in  the 
Bahamas  with  dockside  tests  done  in  Port  Everglades,  Florida. 
Surface  ship  WSAT  inspections  on  the  West  coast  are 
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accomplished  by  technicians  from  Naval  Undersea  Warfare 
Engineering  Station  (NUWES)  detachment  San  Diego,  California 
on  the  San  Clemente  Island  3-D  Underwater  Range  (SCIUR)  with 
dockside  tests  at  Naval  Station  San  Diego  or  Naval  Station 
Long  Beach.  In  the  Hawaii  area,  inspections  are  done  by  NUWES 
detachment  Hawaii  on  the  Barking  Sands  3-D  Tactical  Undersea 
Range  (BARSTUR)  with  dockside  tests  done  at  Naval  Station 
Pearl  Harbor. 

3.  Objectives 

NAVSEA  Publication  OD  40087  [Ref.  l:pp.  1-1],  the  ASW 
WSAT  program  manual  defines  the  objectives  of  the  WSAT  program 
as  follows: 

•  To  ensure  each  surface  ship  ASW  system  reaches  the  fleet 
in  satisfactory  condition  and  is  able  to  meet  its  assigned 
operating  capabilities. 

•  To  identify  system  defects  prior  to  the  expiration  of 
contractual  industrial  warranties. 

•  To  develop  and  analyze  data  on  system  specifications  and 
tolerances  and  refine  the  same. 

•  To  identify  the  appropriate  command  or  agency  responsible 
when  system  discrepancies  degrade  performance. 

•  To  promote  meaningful  system  improvement  programs. 

4.  Inspection  Sequence  Of  Events 

When  the  Type  Commander  requires  a  WSAT  conducted  on 
one  of  his  ships  it  is  accomplished  via  request  through  the 
Fleet  Commander  scheduling  system  in  conjunction  with  the 
responsible  WSAT  agency.  The  responsible  activity  then 


assembles  a  stock  test  plan  for  the  ship  based  on  its 
configuration.  Accordingly,  the  scope  of  the  inspection 
varies,  based  not  only  on  the  ASW  suite  installed,  but  also  on 
the  inspectors  available  at  inspection  time.  The  WSAT 
consists  of  four  inspection  phases  culminated  with  exercise 
weapons  firings  on  an  instrumented  3-D  range  to  evaluate  total 
ASW  system  dynamic  performance.  The  WSAT  consists  of  both 
dockside  and  underway  testing,  evaluation,  and  analysis.  A 
summary  of  each  WSAT  inspection  phase  is  provided  as  follows: 

•  Phase  I  (5-7  Days  Dockside) 

1.  Arrival  conference  and  equipment  set  up. 

2.  ASW  fire  control  operability  and  accuracy  checks. 

3.  Internal  sonar  alignment  checks. 

4.  ASW  system  interface  tests  and  alignments. 

5.  Gyrocompass  and  inertial  navigation  error  tests. 

6.  Onload  of  exercise  weapons  and  range  instrumentation. 

•  Phase  II  (1  Day  Dockside) 

1.  Fire  control  and  torpedo  tube  interface  tests. 

2.  Tube  firing  voltage  and  launch  pressure  tests. 

3.  Dummy  MK-46  torpedo  firings  on  each  tube. 

4.  Measurement  of  weapon  exit  velocity. 

5.  Weapon  recovery,  inspection,  and  preset  verifications. 

•  Phase  III  (1-2  Days  Underway) 

1.  3-D  range  Sensor  Accuracy  Test  (SAT) 

2.  Calibration  of  electromagnetic  log. 

3.  Gyroscope  and  inertial  navigation  heading  checks. 
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•  Phase  IV  (1  Day  Underway) 


1.  Total  system  end-to-end  test  on  3-D  range. 

2.  MK-46  over  the  side  exercise  torpedo  firing. 

3.  Exercise  ASW  rocket  (ASROC)  firing  from  each  launcher. 

4.  Helicopter  exercise  torpedo  firing  (if  so  equipped). 


A  summary  of  WSAT  exercise  firings  is  provided  in  Figure  2. 
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Upon  completion  of  the  WSAT  ,  test  results  are  promulgated 
in  the  following  manner: 

•  WSAT  quicklook  message  report  (24  Hours) 

1.  Identifies  all  major,  uncorrected  discrepancies. 

•  WSAT  final  message  report  (10  Days) 

1.  Identifies  all  remaining  uncorrected  deficiencies. 

2.  Recommends  corrective  action  on  problems. 

3.  Provides  NAVSEA  recommendation  for  certification. 


5.  Record  Keeping 

After  the  WSAT  final  report  message  is  received  by  the 
cognizant  inspection  activity  it  is  tracked  until  major 
discrepancies  prohibiting  certification  are  cleared  (if 
certification  is  not  initially  achieved) .  At  that  time  NAVSEA 
issues  a  certification  statement  for  the  ship  concerned.  Then 
the  inspecting  activity  files  the  hard  copy  of  the  WSAT  final 
report  message  in  its  respective  technical  library.  NAVSEA 
(06UT)  then  utilizes  the  ASW  test  program  data  base  for 
storage  of  pertinent  WSAT  information.  NAVSEA,  NUSC,  and 
NUWES  detachments  are  the  holders  and  users  of  this 
microcomputer  based  data  base.  It  will  be  described  and 
evaluated  in  Chapter  III  of  this  analysis. 


6 


B .  WSAT  PROBLEMS 


1.  Cost 

In  recent  years  funding  limits  have  precluded 
accomplishment  of  WSAT  on  all  required  ships,  particularly 
those  older  ships  exiting  an  industrial  environment. 
Interviews  with  officials  at  NAVSEA  have  clarified  the  policy 
that  all  ships  with  ASW  missions  will  receive  WSAT  or  some 
reduced  scope  facsimile  thereof  rather  than  inspect  just  a  few 
ships  with  a  complete  WSAT.  Currently,  AEGIS  and  BURKE  class 
new  construction  and  SPRUANCE  class  ASW  suite  conversions  are 
receiving  priority  for  funding.  As  a  result  of  funding 
problems,  older, 
less  ASW  oriented, 
ships  such  as  non- 
AEGIS  class 
cruisers  and  PERRY 
class  guided 
missile  frigates 
may  not  be  funded 
for  recertification 
inspections  as  they 
exit  regular 
overhaul  or  other 
such  industrial 
availabilities  . 
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Figure  3.  West  Coast  Cost  Data  (Thousands) 


This  is  of  concern  since  most  of  these  ships  are  tentatively 
scheduled  to  remain  on  the  active  duty  roster  through  the  year 
2000.  Figure  3  provides  fiscal  year  1992  budget  estimates  of 
West  coast  WSAT  costs  based  on  required  inspections  forecast 
in  the  ASW  test  program  data  base.  Force  downsizing  will  be 
providing  relief  with  the  ongoing  decommissioning  of  all  KNOX 
class  frigates  as  well  as  the  COONTZ  and  ADAMS  class  guided 
missile  destroyers.  As  for  the  non-AEGIS  cruisers  of  the 
LEAHY  and  BELKNAP  class  which  will  remain  on  active  duty  in 
the  near  future,  the  April  1992  Navy  Times  [Ref.  2:p.  4] 
outlined  Washington  Navy  proposals  to  possibly  remove  or  at 
least  deactivate  sonar  and  ASW  weapons  systems  from  these 
ships.  This  effort  will  cut  operations  and  maintenance  costs 
and  take  a  step  toward  eliminating  obsolete  systems  from  the 
fleet.  It  would  also  free  up  additional  training  facilities 
ashore  and  provide  cost  savings  in  the  manpower  area.  The 
overall  result  with  respect  to  ASW  test  programs  would  be 
fewer  ships  to  certify. 

2 .  Redundant  Inspections 

An  additional  problem  effecting  complete  funding  of 
required  WSAT  inspections  is  that  of  redundant  inspections. 
The  current  impetus  at  NAVSEA  is  to  eliminate  redundant 
inspections,  or  at  least  to  consolidate  testing  where  possible 
to  promote  greater  fiscal  efficiency.  An  example  of  this  is 
the  AN/SQS-89(V)  surface  ship  ASW  suite  certification.  This 
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initial  acceptance  certification,  which  contains  many  of  the 
same  testing  elements  as  WSAT,  pertains  to  new  ship 
construction  programs  and  is  intended  to  ensure  specifications 
are  met  satisfactorily.  This  program  is  an  important  part  of 
acquisition  quality  assurance  policies  in  the  Navy's  dealings 
with  outside  vendors.  Interview  with  Mr.  Bob  Devon  of 
NAVSEACENPAC  indicates  that  early  timing  of  this  certification 
with  respect  to  crew  training  and  arrival  onboard  from 
precommissioning  sites  may  preclude  substitution  for  WSAT  in 
whole.  However,  some  portions  of  the  testing  are  candidates 
for  consolidation  in  WSAT  testing.  Another  example  of  this  is 
the  ASW  Systems  Qualification  Trials  (SQT)  ,  a  part  of  the 
greater  Combat  System  Shipboard  Qualification  Trials  (CSSQT) . 
Defined  under  NAVSEAINST  9093. 1A,  CSSQT  encompasses  the  entire 
weapons  suite  and  is  heavily  oriented  to  both  operations  and 
maintenance  training.  Although  a  favorite  of  the  fleet  due  to 
the  expansive  scope  of  services,  exercise  shots,  and  training 
opportunities,  NAVSEA  has  emphasized  its  responsibility  for 
material  readiness  and  not  for  training.  Although  no  cost 
figures  were  available  on  CSSQT  or  ASW  SQT,  it  is  certain  that 
costs  are  several  times  that  of  WSAT  alone.  Despite  the  fact 
that  CSSQT  and  ASW  SQT  meet  all  requirements  for  WSAT,  the 
certification  processed  is  subject  to  competing  demands  in 
other  combat  systems  areas.  Likewise,  WSAT  inspection 
scheduling  and  budget  controls  at  NAVSEA  (06UT)  are 
relinquished  under  CSSQT  cognizant  authorities. 
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As  of  January  1992,  CSSQT  and  A3W  SQT  are  unfunded  for  the 
foreseeable  future  with  the  exception  of  lead  ships  in  a  new 
class  such  as  the  BURKE  class  guided  missile  destroyer. 

3.  Reduced  Scope  Inspections 

In  an  effort  to  provide  a  WSAT  for  every  ship  when 
due,  the  current  effort  is  to  develop  a  reduced  scope 
inspection  which  will  meet  certification  requirements. 
Specifically,  the  requirement  for  recertification  after  ROH  or 
DSRA  is  of  interest  because  statistically  these  were  the  ship 
inspections  left  unscheduled  due  to  funding  constraints.  The 
first  endeavor  for  a  reduced  scope  WSAT  inspection  was  the 
Surface  ASW  System  Test  (SAST) .  The  SAST  Program  Manual 
[Ref.  3:p.  1-2]  outlines  the  scope  of  the  inspection.  Figure 
4,  taken  from  that  manual,  displays  the  proposed  relationship 
of  the  SAST  program  to  the  ship  operational  life  cycle. 


Figure  4.  Ship  Life  Cycle  (SAST) 
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The  first  SAST  inspection  was  accomplished  in  January  1992  at 
Long  Beach  Naval  Station,  California  aboard  the  USS  PAUL  F. 
FOSTER  (DD-964 ) .  The  SAST  inspection  was  designed  to  cost 
approximately  one  third  that  of  a  WSAT  and  provide  a 
reasonable  assurance  that  the  ASW  combat  system  is  ready  for 
underway  operations  and  meets  requisite  performance 
specifications.  Although  the  four  phase  SAST  inspection  is 
similar  to  WSAT  on  paper,  some  significant  differences  exist. 
First,  a  shift  of  emphasis  from  outside  source  technician 
testing  to  ship's  force  requires  that  the  ship  complete  Phase 
I  dockside  testing  prior  to  the  arrival  of  NUWES/NUSC  ASW 
system  experts  in  phase  II.  As  a  result,  ship's  force  PMS 
checks  now  form  the  basis  for  continued  testing.  This 
assessment  is  performed  by  the  SAST  team  leader.  This  concept 
removes  the  necessity  of  a  WSAT  team  from  being  on  travel  for 
the  usual  WSAT  phase  I  (5-7  days)  dockside  tests. 
Unfortunately,  it  shifts  the  foundation  for  ASW  certification 
to  the  command  involved  without  regard  to  current  ship  manning 
and  training  levels  subsequent  to  an  industrial  availability 
when  they  are  historically  under  strength.  As  a  result  some 
material  categories  could  receive  little  or  no  attention  by 
the  more  qualified  NUSC/NUWES  engineers  in  phase  II,  based  on 
erroneous  testing  by  ships  force  early  in  phase  I. 
Additionally,  the  phase  IV  WSAT  3-D  range  firing  exercises  are 
eliminated.  This  concept  eradicates  the  total  system  end-to- 
end  test  deemed  important  by  the  fleet  as  well  as  test 
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engineers.  Also,  it  is  unclear  if  anywhere  in  SAST  an  effort 
is  made  to  scientifically  determine  where  to  concentrate 
scarce  inspection  dollars  based  on  material  discrepancy  trends 
or  histories.  The  latest  endeavor  at  NAVSEA  in  restructuring 
WSAT,  currently  in  the  planning  stage  only,  is  referred  to  as 
the  Consolidated  ASW  Test  (CAT)  program.  The  current  surface 
ship  ASW  test  plan  POA&M  is  shown  in  Figure  5. 

SURFACE  SHIP  ASW  COMBAT  SYSTEM 
CERT  PROGRAM  POA&M 

DEC  91  Establish  mimimum  trial  requiremsnts 

JAN  92  Define  dockside  requirements  (Pt  1,  Pt  2) 

Underway  requirements 

FEB  92  Prepare  typical  PMS  test  pkQ  (tor  dockside  Pt  1) 

MAR  92  Review  PMS  test  pkg  (TYCOMs.  NAVSEA) 

APR  92  Prelim  test  plan  &  daily  schedule 

MAY  92  Review  test  plan/schedule  (by  TYCOMs,  NAVSEA) 

JUN  92  Final/approved  test  plan/schedule 

JUL  92  Schedule  first  ship  for  CAT  trials 

AUG  92  Brief  first  ship  on  CAT 

OCT  92  Conduct  first  CAT  trials 

OCT  92/  Monitor,  evaluate  effectiveness  of  CAT 
DEC  92  Propose  revisions  and  improvements 


Figure  5.  ASW  Test  Program  POA&M 
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The  goal  of  the  CAT  program  appears  similar  to  SAST  in  that  it 
will  define  the  post-shipyard  ASW  combat  system  certification 
program  for  surface  ships.  The  program  will  meet  OPNAV 
requirements  to  consolidate  redundant  inspections  as  well  as 
address  fleet  concerns  for  collateral  training  opportunities 
alongside  NUSC/NUV7ES  technicians  in  both  operations  and 
maintenance.  The  process  is  currently  under  Total  Quality 
Leadership  (TQL)  review  to  provide  quality  performance 
indicators  which  measure  effectiveness  and  ensure  ongoing 
program  improvement.  The  lessons  learned  from  the  only  SAST 
inspection  performed  will  be  incorporated  by  increasing 
NUSC/NUWES  supervision  during  phase  I  dockside  testing.  As  in 
SAST,  increased  reliance  on  ships  force  personnel  in 
conducting  selected  maintenance  checks  will  be  utilized.  This 
program  will  attempt  to  incorporate  the  existing  ship  PMS 
schedule  with  results  from  applicable  shipyard  testing  to 
reduce  or  eliminate  redundant  and  excessive  testing.  However, 
the  program  core  testing  requirements  do  not  specify  any 
analysis  of  systems  discrepancy  histories  in  regard  to 
constructing  individual  ship  test  packages.  Similar  to  SAST, 
elimination  of  3-D  range  torpedo  exercises  are  sacrificed  in 
lieu  of  simulated  ASROC,  OTS,  and  LAMPS  exercises. 
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4.  Ramifications  and  Concerns 

The  overall  thrust  of  the  new  ASW  systems  test  program 
appears  to  make  the  best  of  a  bad  situation  with  regard  to 
budget  shortfalls.  Funding  of  testing  for  new  construction 
BURKE  class  guided  missile  destroyers  will  undoubtedly  be 
preserved,  the  problem  lies  with  recertifying  the  aging 
SPRUANCE  and  PERRY  class  ships.  Likewise,  the  AEGIS  cruisers 
of  the  TICONDEROGA  class  will  also  require  some  type  of 
recertification  inspection  as  they  progress  through  their  life 
cycle.  The  last  ship  of  this  class  was  commissioned  in  July 
1992.  The  shifting  of  testing  prerequisites  to  ships  force  as 
a  prelude  to  ASW  certification  will  pose  another  scheduling 
problem  for  afloat  commands.  This  problem  can  be  minimized  by 
adequate  involvement  of  the  type  commander  and  NAVSEA  when 
planning  ship  overhauls.  Additional  efforts  on  the  part  of 
the  ship  will  be  required  to  ensure  a  core  of  sufficiently 
trained  technicians  are  onboard  at  certification  time.  This 
will  minimize  possible  compromise  of  initial  phase  dockside 
testing  by  unqualified  technicians  under  pressure  to  meet 
schedule  deadlines.  A  resounding  need  for  ongoing  statistical 
analysis  in  material  category  reliability  is  called  for  to 
prevent  planned  testing  initiatives  from  risking  overlook  of 
problematic  areas  in  construction  of  a  ship  test  package. 


C.  THESIS  OBJECTIVES 

1.  Material  Discrepancy  Data  Analysis 

With  the  substantial  data  available  on  WSAT  ASW  combat 
system  material  discrepancies,  the  first  objective  of  this 
thesis  is  to  perform  a  categorical  data  analysis  to  reveal 
which  material  categories  are  most  prone  to  failure.  This 
would  refute  an  implicit  assumption  that  all  material 
categories  are  uniformly  degraded  and  that  reduced  scope 
inspections  need  not  statistically  consider  system  material 
histories  during  program  conception. 

2.  WSAT  Data  Base  Examination 

The  second  objective  of  this  thesis  will  be  a  critical 
examination  of  the  existing  WSAT  data  base.  This  examination 
will  include  recommendations  for  future  modifications  to  the 
data  base;  primarily  with  respect  to  the  deletion  and  addition 
of  specific  data  fields.  These  recommendations  will  be  biased 
toward  what  will  be  required  to  make  the  data  base  a  viable 
tool  in  future  statistical  analysis. 
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II.  WEAPON  SYSTEM  ACCURACY  TRIAL  (WSAT)  DATA 


A.  DATA  COLLECTION 

1.  Experience  Tour 

During  the  six  week  ASW  curriculum  experience  tour  in 
quarter  5  of  8,  several  invaluable  point  of  contacts  were  made 
to  facilitate  WSAT  thesis  data  collection.  Mr.  Edward  Biery 
of  NUWES  detachment  Keyport,  WA.  (code  56)  was  sponsor  of  the 
experience  tour.  It  was  his  office  from  which  most  background 
information  on  the  WSAT  program  was  obtained.  That  office 
arranged  a  visit  to  NAVSEA  (06UT),  Washington,  D.C.  to  meet 
senior  civilian  engineers  in  the  ASW  test  program  as  well  as 
STGCS(SW)  W.J.  Vick,  developer  of  the  NAVSEA  ASW  test  program 
data  base.  During  this  visit  NAVSEA  policies  and  goals  with 
respect  to  the  WSAT  program  were  discussed  and  clarified. 

2.  Data  Sources 

The  actual  data  for  use  in  this  analysis  was  obtained 
from  various  subordinate  activities  to  NAVSEA.  Specifically, 
the  data  was  obtained  in  naval  message  format  from  the 
following  activities: 

•  Pacific  Fleet  Data 

1.  NUWES  detachment,  San  Diego,  CA.  (code  906) ,  Mr.  Alex 

Rios  (Head  ASW  Analysis  Division) . 

2.  NUWES  detachment,  Hawaii  (code  90),  Mr.  Adolph  Neumann 

(Director  Pacer) . 
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•  Atlantic  Fleet  Data 


1.  NUSC  Newport,  R.I.  (code  38),  Mr.  John  Peters  (ASW 
Test  Analyst) 


A  summary  of  the  WSAT  data  received  for  this  analysis  is 
provided  in  Figure  6. 


WSAT  DATA  SOURCES 
BY  SHIP  TYPE 

SHIP  TYPE  BY  CLASS  NUMBER  OF  SHIPS  TESTED 

GARCIA  (FF) 

i 

KNOX  (FF) 

9 

USCG  (WHEC) 

4 

BROOKE  (FFG) 

2 

PERRY  (FFG) 

13 

SPRUANCE  (DD) 

16 

KIDD  (DDG) 

3 

BURKE  (DDG) 

1 

LEAHY  (CG) 

2 

TICONDEROGA  (CG) 

13 

TOTAL  NUMBER 

OF  SHIPS  TESTED 

64 

SURFACE  SHIP  WSAT  DATA  (5  YEARS) 

Figure  6.  WSAT  Data  Sources 
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B.  DATA  ORGANIZATION 


1.  Data  Specifics 

A  total  of  64  WSAT  final  inspection  report  messages 
covering  the  last  five  years  were  obtained  from  cognizant 
inspection  activities  for  use  in  this  analysis.  The  data 
represents  both  Pacific  and  Atlantic  fleet  surface  assets 
which  have  ASW  as  one  of  the  command  primary  warfare  areas. 
The  last  five  years  of  data  was  chosen  for  this  analysis: 

1.  To  minimize  interruption  of  business  at  the  various 
inspection  activities  who  were  called  on  to  locate, 
reproduce,  and  ship  the  data. 

2.  To  minimize  wasteful  receipt  of  older  data  which  is 
representative  of  ships  scheduled  for  upcoming 
decommissioning . 

The  64  final  report  messages  contain  a  total  of  595  inspection 
discrepancies.  These  data  elements  chosen  for  analysis  are  a 
representative  population  sample  of  all  WSAT  discrepancies. 
Each  data  element  represents  a  failed  maintenance  inspection 
or  operational  check.  For  purposes  of  the  categorical  data 
analysis  to  be  discussed  in  Chapter  IV,  each  check  is  the 
result  of  a  failed  binomial  experiment.  The  sampled  data  was 
further  assigned  to  one  of  five  specific  material  categories. 
The  categories  were  not  chosen  arbitrarily,  but  rather  were 
chosen  per  the  prescribed  categorical  reporting  format  of  the 
final  report  message  described  in  NAVSEA  OD  40087.  With  all 
inspection  reports  describing  both  major  and  minor 
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discrepancies  in  this  manner,  aggregation  of  all  data  elements 
into  appropriate  categories  became  a  less  arduous 
administrative  task  in  bookkeeping. 

2.  Data  Categories 

To  clarify  which  material  discrepancies  belong  to 
specific  categories,  a  breakdown  is  summarized  as  follows: 

•  Category  I  (Sonar  Systems) 

1.  Hull  mounted  sonar  systems 

2 .  Towed  array  sonar  systems 

3.  Signal  processing  systems 

•  Category  II  (ASW  Aviation  Systems) 

1.  Sonobuoy  systems 

2.  ASW  helicopter  data  link  systems 

3.  Torpedo  loading,  arming,  and  presetter  systems 

•  Category  III  (Weapons  Delivery  Systems) 

1.  Surface  vessel  torpedo  tubes,  all  makes  and 
modifications 

2 .  Asroc  box  launcher  systems 

3.  Asroc  guided  missile  launcher  systems 

4.  Asroc  vertical  launch  systems 

•  Category  IV  (ASW  Fire  Control  Systems) 

1.  The  Mk-114  analog  fire  control  system 

2.  The  Mk-116  digital  fire  control  system 

3.  The  USCG  Mk-309  fire  control  system 
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•  Category  V  (ASW  Sensor  Subsystems) 

1.  The  ship  electromagnetic  log 

2.  Ship  gyroscope  and  inertial  navigation  systems 

3.  ASW  related  command  decision  systems  (CDS/NTDS) 

4.  ASW  related  gunfire  control  radars  and  systems 

5.  ASW  related  consoles  and  repeaters 

C.  DATA  SUMMARY  PRESENTATION 

For  presentation  of  the  data  by  categories  bar  graphs  were 
chosen  for  relative  comparisons  between  material  categories. 
For  this  analysis  the  data  is  broken  down  by  Atlantic  and 
Pacific  fleet  and  also  shown  Navy  wide  in  Figures  7,  8,  and  9. 


ATLANTIC  FLEET  WSAT 
DISCREPANCIES  BY  CATEGORY 


■  M  *0  TO  97  M 

SURFACE  SHIP  WSAT  DATA  (5  YEARS) 

Figure  7.  Atlantic  Fleet  WSAT  Data 
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Figure  8.  Pacific  Fleet  WSAT  Data 
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NAVY  WIDE  WSAT 
DISCREPANCIES  BY  CATEGORY 

200 

150 

100 

50 

0  SONAR  SYS.  AVIATION  SYS.  WEAPONS  SYS.  P.C.  SYS.  SENSOR  SUBSYS. 

■  117  M  168  130  128 

SURFACE  SHIP  WSAT  DATA  (5  YEARS) 


Figure  9.  Navy  Wide  WSAT  Data 
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III.  THE  WSAT  DATA  BASE 


A.  DATA  BASE  RESPONSIBILITY 

The  WSAT  data  base  is  managed  by  the  NAVSEA  ASW  test 
program  office  (NAVSEA  06UT) .  It  is  the  product  of  efforts  by 
STGCS(SW)  W.J.  Vick,  USN .  It  was  designed  to  be  an  all 
inclusive  information  system  containing  relevant  historical 
information  concerning  ASW  testing  conducted  by  NAVSEA  on  both 
surface  ships  and  submarines.  Additionally,  it  is  intended  to 
be  used  as  an  aid  in  forecasting  future  testing  requirements 
based  on  long  range  ship  repair  and  conversion  plans.  The 
information  in  the  centralized  data  base  is  compiled  from  many 
different  sources.  The  data  base  is  currently  managed  by  Mr. 
Peter  Karounos  (NAVSEA  06UT)  who  is  assisted  by  STGCS(SW)  J. 
Brown,  USN.  As  a  centralized  data  base,  NAVSEA  retains 
responsibilities  for  both  data  integrity  and  security. 

B.  ORGANIZATION 

The  latest  printed  edition  of  the  WSAT  ASW  test  data  base 
is  dated  16  September  1991  and  is  organized  into  two  parts. 
Part  one  is  information  relevant  to  surface  ships.  Part  two 
is  information  relevant  to  submarines.  Further  discussion 
will  be  limited  to  the  surface  ship  section  of  the  data  base 
in  accordance  with  the  scope  of  this  analysis.  As  a 
microcomputer  based  software  data  base  management  system 
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written  on  the  commercially  available  PFS:Professional  File 
Version  2.02,  the  data  base  is  easily  learned  and  manipulated. 

G.  Hanson  and  J.  Hanson  in  Database  Management  and  Design 
[Ref.  4:pp  15]  describes  a  microcomputer  data  base  management 
system  as  a  system  that  demonstrates  query  flexibility  and 
ease  of  data  base  creation  and  maintenance.  This  package 
meets  those  prerequisites.  The  surface  ship  data  base 
currently  comprise  51  data  fields  of  various  types  (ie:  date, 
numeric,  alphanumeric,  etc...).  Figure  10  displays  the 
relevant  data  fields  being  used. 

SURF  Inter  instructions  tor  printing  nailing  labels.  Page  1  of|2 

Fl-Help  F2 -'Label  F3-Zdit  F4 -Quick  Entry  FlO-Continue 


HULL: 
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uic: 

CLASS: 

CLASS  NAME: 

TYCOM:  PAC: 

LANT: 

GRP: 

SQN: 

PORT: 

FY:  QTR: 

PRI: 

CERT: 

SUPPORT: 

AVAIL: 

FUND: 

YARD: 

END 

YARD: 

LAST  WSAT: 

LAST  DATE: 

SCOT: 

LIM  DIT: 

CSSQT; 

LSQT: 

SAT  RPTi 

SAT: 

SAT 

DATE: 

SAST: 

TL: 

PHONE: 

SAST  LTR: 

♦M  DEF: 

InDEr: 

WSAT: 

TCA: 

TDA: 

PHONE: 

QUICK  LOOK: 

DOCKSIDE: 

RANGE: 

FINAL  REPORT: 
NOTES: 

MAJOR: 

MINOR: 

CLEARED:  LEFT: 

WITHHOLD  LTR: 
CERT  LTR: 
REMARKS: 

Figure  10.  WSAT  Data  Base  Fields 
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C.  CRITICAL  EXAMINATION 

PFS: Professional  File  is  a  flexible  software  application 
package  which  is  very  similar  to  Data  Base  III.  The 
comprehensive  use  of  pull  down  menus  and  on  line  help 
functions  make  it  quite  adaptive  to  the  novice  user,  yet  it 
also  has  many  short  cut  features  suited  to  the  advanced 
operator.  Although  the  software  system  is  more  than  adequate 
for  the  job,  the  data  base  itself  has  several  aspects  needing 
revision  to  ensure  continued  usefulness.  Critical  examination 
indicates  that  while  a  basis  for  corporate  information  system 
management  is  present,  approximately  8  data  fields  in  the  data 
base  are  either  obsolete  or  unnecessary.  This  statement  is 
substantiated  by  the  appalling  lack  of  data  in  these  fields 
and  the  insignificance  of  the  information  in  others.  A 
complete  printed  report  of  the  entire  data  base  requires  4 
pages  to  intelligently  convey  all  information  on  each  ship. 
It  is  obvious  upon  examination  that  the  data  base  is  primarily 
an  administrative  aid  to  track  ASW  testing,  scheduling,  and 
associated  correspondence.  Unfortunately,  the  usefulness  of 
the  data  base  in  identifying  material  discrepancy  trends  is 
limited  in  the  current  format.  Although  the  number  of 
discrepancies  are  listed  for  each  ship  inspection,  it  fails  to 
identify  equipment  specifics  or  even  the  associated  material 
categories.  This  information  will  be  statistically 
significant  in  planning  future  inspections  or  other 
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certification  procedures.  This  statement  is  even  more  true  in 
a  time  of  budget  shortfalls  and  constraints,  which  appear  to 
be  a  permanent  fixture  associated  with  the  conclusion  of  the 
cold  war. 

As  future  certification  procedures  evolve,  probably  based 
on  cost  overhead,  it  would  be  imprudent  to  ignore  material 
trends  where  risks  resulting  from  non-testing  are  too  great. 
The  problem  of  manpower  in  maintaining  the  ASW  test  data  base 
might  be  addressed  by  changing  from  a  centralized  data  base 
management  system  to  a  distributed  system  connected  via  modem. 
This  would  spread  the  chores  of  data  entry  to  the  requisite 
inspection  activities  and  put  NAVSEA  in  a  more  supervisory 
role.  The  network  could  conceivably  consist  of  only  5  sites 
which  is  easily  manageable.  With  the  unclassified  nature  of 
the  data  base  and  the  use  of  commercial  phone  lines,  this 
concept  is  certainly  economically  feasible.  Specific 
recommendations  on  data  base  field  deletions  and  additions 
will  be  included  in  Chapter  VI  of  this  study. 
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IV.  ANALYSIS  OF  DATA 


A.  ANALYSIS  ASSUMPTIONS 

As  the  scope  of  the  ASW  test  and  certification  program, 
specifically  WSAT,  undergoes  dynamic  changes  to  meet  funding 
restrictions,  the  question  of  which  material  categories 
historically  demonstrate  the  most  discrepancies  should  be 
examined.  If  the  WSAT  core  structure  inspection  is  reduced 
without  regard  to  this,  then  it  is  an  implicit  assumption  that 
material  degradation  is  homogenous  throughout  the  five 
material  categories  in  the  WSAT.  Although  WSAT  describes 
inspection  discrepancies  as  either  major  or  minor  in  nature, 
this  analysis  will  weigh  all  discrepancies  equally  for 
conducting  the  multinomial  experiment.  Accordingly  our  N  = 
595  data  points  will  be  individually  assigned  to  one  of  five 
material  categories.  The  variable  pi  will  represent  the 
probability  that  any  individual  discrepancy  belongs  in 
material  category  i  (i=l . 5). 

A.  NULL  AND  ALTERNATE  HYPOTHESIS 

In  order  to  conduct  a  statistical  test  on  the  data,  a 
specific  hypothesis  to  be  refuted  must  be  defined.  For  this 
analysis  the  null  hypothesis  will  be  that  any  particular 
observation  is  equally  likely  to  fall  in  any  one  of 
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the  five  WSAT  material  categories  described  in  chapter  II, 
that  is  P/=P2=Pj=P<=Pj*  The  alternate  hypothesis  to  be 
substantiated  via  testing  is  that  at  least  one  category 
proportion  is  not  equal  to  the  others.  Equation  1  summarizes 
the  null  hypothesis: 


NULL  HYPOTHESIS 
Ho  :  Pi  =  Pi  =  Pi  =  Pa  =  Ps  =  20 

Equation  2  summarizes  the  alternate  hypothesis: 


(1) 


ALTERNATIVE  HYPOTHESIS 
//  :  At  least  one  p  *  .20 

a  ‘  i 


B.  STATISTICAL  TESTING  OF  HYPOTHESIS 

A  standard  test  procedure  for  this  hypothesis  is  described 
in  Devore,  Probability  And  Statistics  For  Engineering  And  The 
Sciences  [Ref.  5:pp  556-559].  The  name  of  this  procedure  is 
the  chi-squared  goodness-of-f it  test  for  categorical  data. 
The  null  hypothesis  is  rejected  in  favor  of  the  alternative 
hypothesis  if  a  test  statistic  is  too  large.  The  test 
statistic  is  labeled  a  x:  (chi-square)  statistic  and  is 
defined  in  Equation  3 


x2  =  E 


(«,  -  *)2 


(3) 
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where  n,  is  the  number  of  the  total  N  =  595  observations  that 
fall  into  each  material  category  i  (i=l, 2 , 3 , 4 , 5)  found  in 
Chapter  II,  Figure  9.  The  variable  e,  is  the  expected  number 
of  the  595  observations  for  each  category  i  as  calculated  in 
Equation  4 . 

=  P,(N)  =  (20)(595)  =  119  (4) 


When  the  null  hypothesis  is  true,  the  observed  number  of 
category  discrepancies  should  be  close  to  the  corresponding 
expected  values  and  the  test  statistic  should  be  small. 
Provided  that  the  expected  value,  e,,  is  greater  than  or  equal 
to  5  for  each  category,  the  test  statistic  has  a  chi-squared 
distribution  with  v  =  5-1  =  4  degrees  of  freedom.  The 
decision  to  accept  or  reject  the  null  hypothesis  will  be 
accomplished  by  comparing  the  computed  value  of  the  test 
statistic  to  an  appropriate  critical  value  of  the  chi-squared 
distribution,  which  is  tabulated  and  available  in  most 
statistics  text  books.  The  critical  value  for  the  test 
procedure  will  be  determined  by  choosing  a  significance  level, 
a  ,  for  the  hypothesis  test.  The  value  of  a  is  the 
probability  of  rejecting  the  null  hypothesis  when  it  is  true. 
The  critical  value  is  the  1-a  percentile  point  of  the  chi- 
square  distribution  with  4  degrees  of  freedom.  For  a  =  .01  and 
v  =  4  the  tabulated  critical  value  is  13.277. 
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This  means  we  reject  our  null  hypothesis  if  the  calculated 
test  statistic  is  greater  than  13.277.  Equations  5  through  7 
summarize  the  Navy  wide  chi-square  test  respectively. 


Reject  Hg  (pt  =.20)  if 


x2  *  x2fl 


(5) 


The  xz  test  statistic  defined  by  equation  (3) 

2  _  (117-119)2  t  (56-119)2  |  (163-119)2 
X  119  +  119  +  119  (Q 

,  (130-119)2  ,  (129-119)2  _  5151 
119  119 


The  tabular  distribution  critical  value  is 
X201>4  =  13.277 
as  a  result,  since 

X2=  51.51^  x20I.4=  13.277 - Reject  Ho 


The  conclusion,  based  on  this  test  procedure,  is  to  reject  the 
null  hypothesis  of  homogenous  material  category  degradation  in 
favor  of  the  alternate  hypothesis.  There  is  very  strong 
statistical  evidence  indicating  significant  differences  in 
Navy  wide  WSAT  material  category  reliability  to  support  the 
alternative  hypothesis. 

The  next  step  in  the  analysis  will  be  to  determine  if  the 
Atlantic  Fleet  data  refutes  the  same  homogenous  null 
hypothesis.  Using  observed  discrepancy  values  from  Chapter 
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II,  Figure  7,  the  pertinent  variables  are  summarized  in 
Equation  8  through  10. 


N  =  261  p.  =  .20  e.  =  (.20)(261)  =52.2  (8) 


The  x2  test  statistic  is 
2  =  (55-52.2)2  t  (30-52.2)2  t  (70-52.2)2 
X  52.2  52.2  +  52.2  W 

+  (67-52.2)2  t  (39-52.2)2_  23  2Q 
52.2  52.2 


X2=23.20  ;>  x2om=13-277  - ^eyecf  Ho  (10) 


Accordingly,  testing  of  the  Atlantic  Fleet  data  also  supports 
rejecting  the  null  hypothesis  in  favor  of  the  alternative 
hypothesis  that  at  least  one  p,  is  not  equal  to  .20  . 

The  last  step  in  the  test  will  be  to  check  that  the 
Pacific  Fleet  data  also  refutes  the  null  hypothesis.  Using 
the  data  from  Chapter  II,  Figure  8  the  variables  for  the  test 
are  summarized  in  Equations  11  through  13. 

N  =  334  Pi  =  .20  e,  =(.20)(334)  =  66.8  (11) 

The  x2  test  statistic  is 
2  =  (62-66.8)2  ^  (26-66.8)2  (  (93-66.8)2 
X  66.8  +  66.8  +  66.8  <12> 

+  (63-66.8)2  ^(90-66.8)2  =  43  g2 
66.8  66.8 
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x2  =  43.82  *  X2 


=  13.277 


(13) 


01,4 


- Reject  H0 


Like  the  Navy  wide  data  and  the  Atlantic  Fleet  data,  the 
Pacific  Fleet  data  also  strongly  supports  rejecting  the  null 
hypothesis  in  favor  of  the  alternative  hypoyhesis. 

Having  tested  the  data  in  the  aforementioned  fashion,  it 
is  now  an  accurate  statement  to  say  that  a  homogenous 
degradation  in  the  five  material  categories  of  WSAT  is  false. 
Chapter  V  will  address  specific  recommendations  on  which 
category  to  focus  attention  on  and  which  categories  are  least 
prone  to  failure. 


32 


V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  WSAT  ANALYSIS 
1.  Conclusions 

The  analysis  of  the  last  five  years  of  WSAT  data 
conducted  in  Chapter  IV  strongly  indicates  the  variation  in 
degradation  among  the  five  material  categories  which  comprise 
WSAT.  Figures  11  through  13  summarize  the  experiment  results 
for  the  Atlantic  Fleet,  Pacific  Fleet,  and  Navy  wide  data 
respectively. 
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FREQUENCY  DISTRIBUTION 
NAVY  WIDE  OBSERVATIONS 
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Figure  13.  Navy  Wide  Results 
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The  Navy  wide  material  categories  rank  from  least  prone  to 
failure  to  most  prone  to  failure  as  follows: 

•  NAVY  WIDE  RELIABILITY  RANKING 

1.  Aviation  Systems  (Cat  II) 

2.  Sonar  Systems  (Cat  I) 

3.  Sensor  Subsystems  (Cat  V) 

4.  Fire  Control  Systems  (Cat  IV) 

5.  Weapons  Delivery  Systems  (Cat  III) 

The  discrepancy  most  common  in  the  least  reliable  category  was 
undeniably  found  to  be  in  torpedo  launcher  systems.  Improper 
tube  firing  voltages,  firing  pressures,  and  interface  failures 
were  prolific  in  the  163  data  points  in  this  category.  This 
is  not  surprising  considering  the  fact  that  most  torpedo 
launcher  systems  have  at  least  partial  exposure  directly  to  an 
ocean  environment  with  detrimental  effects  on  electrical  and 
mechanical  components.  Figure  14  provides  for  direct 
comparison  between  Atlantic  and  Pacific  Fleet  observed 
discrepancies.  In  3  of  5  categories  the  Pacific  Fleet  ASW 
systems  demonstrate  a  greater  propensity  for  failure  than  the 
Atlantic  Fleet.  The  largest  variation  found  was  in  category 
V  (Sensor  Subsystems)  where  the  Pacific  Fleet  showed 
significantly  higher  numbers  of  discrepancies  than  its 
contemporary  (90  vice  39)  .  Subsequent  reexamination  of  the 
data  revealed  a  prevalence  of  discrepancies  in  NTDS/CDS 
software  on  Pacific  Fleet  Ticonderoga  class  guided  missile 
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cruisers.  These  most  recently  delivered  ships  contain  the 
most  current  software  versions  which  apparently  are  replete 
with  certification  discrepancies.  The  other  differences  in 
Fleet  data  are  small  in  comparison  but  have  no  apparent 
rational  explanation.  One  would  expect  that  uniform 
technician  training  standards  and  maintenance  procedures  would 
minimize  this  variance.  The  only  other  possible  explanation 
might  be  a  function  of  East  Coast  and  West  Coast  Test  Activity 
inspector  personalities  and  background  experience. 


FREQUENCY  DISTRIBUTION  COMPARISON 
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2 .  Recommendations 


The  following  recommendations  apply  to  the  future  of 
the  ASW  Test  Program: 

1.  Focus  future  inspection  packages  with  emphasis  on  torpedo 
tube  and  ASROC  launcher  validation  by  experienced 
technicians . 

2.  Establish  revised,  precise  tactical  digital  standards  for 
certification  of  software  based  systems. 

3.  If  3-D  range  firing  exercises  must  be  eliminated,  conduct 
a  comprehensive  study  of  past  range  data  to  determine 
failure  cause  and  results.  Incorporate  these  findings  in 
test  program  development. 

4.  Include  any  available  ASW  PACER  data  in  development  or 
assessment  of  a  ship  ASW  certification. 

5.  Conduct  ongoing  statistical  analysis  of  ASW  combat 
systems  discrepancy  trends.  The  CASREP  data  base, 
maintained  at  the  TYCOM  level,  is  an  excellent  source  of 
current  Fleet  system  problems. 

6.  Incorporate  future  WSAT  (or  equivalent)  data  into  ongoing 
statistical  analysis. 

7 .  Conduct  a  study  to  compare  WSAT  historical  trend  results 
with  current  Fleet  equipment  reliability  prediction 
programs  such  as  the  Trouble  Systems  Process  developed  by 
Vitro  Corporation. 

8.  Conduct  a  comparative  study  of  ASW  technical  training 
standards  between  Fleet  training  centers  with  a  goal  of 
uniformity. 


B.  WSAT  DATA  BASE  ANALYSIS 
1.  conclusions 

The  critique  of  the  existing  WSAT  data  base  in  Chapter 
II  found  a  comprehensive,  centralized  data  base  requiring  only 
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minor  modification.  The  key  issues  are  to  examine  the 
implementation  of  a  distributed  network  data  base  and 
subsequent  modification  of  data  base  fields  to  promote 
statistical  analysis. 

2.  Recommendations 

The  following  recommendations  apply  to  data  base  field 
structure: 

•  Fields  Recommended  For  Deletion 

1.  GRP/SQN/PORT:  All  of  these  fields  are  variable  over  a 
ships  life.  They  are  available  elsewhere  and  actually 
provide  little  information  relative  to  the  ASW  Test 
Program. 

2.  CSSQT:  This  is  no  longer  a  funded  program. 

3.  SCOT:  This  field  contains  minimal  existing  data  which 
attests  to  its  invalidation. 

4.  SAST/SAST  LTR:  Only  one  inspection  was  conducted  under 
this  program  prior  to  discontinuation. 

5.  TDA/ PHONE  #:  This  field  also  has  no  existing  data. 
Test  agency  phone  numbers  are  variable  as  shore 
commands  evolve  and  are  available  elsewhere. 

6.  REMARKS:  This  field  exhibits  minimal  existing  data 
which  attests  to  its  invalidation. 


•  Fields  Recommended  For  Addition 

1.  A  material  category  discrepancy  breakdown  field  which 
would  provide  an  adequate  means  of  recording  observed 
discrepancies  in  an  orderly  fashion  and  promote 
statistical  analysis.  The  following  numeric  field 
format  would  suffice:  — / — / — / — / — . 
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